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ABSTRACT 

This paper deals with the evaluation of an 
out-of -school tutorial program, CROSSROADS, which was funded under 
the U.S. Department of Education's Basic Skills Improvement Program. 
Like many projects funded under the Basic Skills program, CROSSROADS 
was terminated after two years even though it had been funded with 
the expectation of four years for program develoinnent and 
demonstration. Subjects of the evaluation were 166 tutees from four 
junior high schools in Long Beach, California. Progress rates of the 
tutees were measured by pre- and posttests, standardized tests, 
school grades and criterion referenced measures. Evaluation results 
suggested that tutees learned some basic skills but not enough to 
impact upon standardized test scores or grades. Results also 
suggested that out-of-school basic skills programs will have a 
difficult time in producing substaiitial gains in basic skills in 
lower achieving children due to problems with tutee motivation and 
logistics of running out-of-school programs. Although the evaluation 
did not substantiate strongly the effectiveness of the CROSSROADS 
model for basic skill improvement, findings and issues are pertinent 
to efforts to improve basic skills and to educational policies and 
evaluation practice. (Author /JAZ) 
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Abstract 

This paper deals with the evaluation of an out-of -schoo 1 
tutorial program, CROSSROADS, which was funded under the U.S. 
Department of Education's Basic Skills Improvement Program. 
Like many projects funded under the Basic Skills program. 
CROSSROADS was ter m inated after two years even though it had been 
funded with the expectation of four years for program development 
and demonstration. Evaluation results suggested that tutees 
learned some basic skills but not enough to impact upon 
standardized test scores or grades. Results also suggested that 
out-of -schoo 1 basic skills programs will have a difficult time in 
producing substantial gains in basic skills in lower achieving 
children due to problems with tutee motivation and logistics of 
running out^of -schoo 1 s programs. Although the evaluation did not 
substantiate strongly the effectiveness of the CROSSROADS model 
for basic skill improvement, findings and issues are pertinent to 
efforts to improve basic skills and to educational policies and 
evaluation practice- 
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In 1980, the U.S. Depar ttnent*'6'f Education launched the 
Basic Skills Improvement Program, About one hundred and tifty 
demonstration projects were runded nationwide at a time or 
national concern about, railing achievement test scores. 
The Department or Education encouraged a broad range or 
approaches to basic skills improve ment in three general 
categories- in-school projects* out-'of -schoo i projects, and 
parent involvement projects. The Basic Skills Improvement 
Program was terminated after two years even though many projects 
had been funded with the expectation of four years of funding for 
program development and demonstration. This paper deals with the 
evaluation of an out-of -schoo 1 tutorial program* CROSSROADS. 
Although the evaluation did not strongly substantiate the 
ef f ect i ^'eness of the CI^OSSROADS model for basic skill 
improvement, findings and issues are pertinent to efforts to 
improve basic skills and tc educational policies and educational 
eva 1 uat i on pr ac t i ce . 

That tutoring can be effective seems to be well-established 
based upon the accumulation of evaluative literature 
(Fi tz-Gibbons , 1979). This extends to the conclusion that 
tutoring can be more effective than ordinary classroom 
instruction CBausell, Moody, and Walzl, 1972). Part of the 
importance or tutoring lies in t.he tutor--tutee reiationsnip 
(Fi tz-Gibbons, 1979>. if a lower achieving tutee works with a 
higher achieving tutor, the tutee may identify with the tutor, 
raising the child's academic se 1 f -concept , educational confidence 
and aspirations- This process might be especially important for 
families lacking a role model of academic success. 
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CROSSROADS 

Given this knowledge* one reasonable approach to basic 
skills improvement in 1980 was to develop a supplemental tutoring 
program ror children at the lowest end of the basic skills 
achievement spec^irum. CROSSROADS had a fairly complex impact 
model, or plan for how program activities would achieve program 
objectives and goals. This impact model had five general 
components including tutee variables, environmental conditions, 
family variables, program services, and proximal and distal 
outcomes. Each of the program services was based upon one or more 
aspect of the impact model. 
PROGRAM SERVICES 

TUTORING . The tutoring model was sin individualized, highly 
structured approacn using paid college students as tutors. The 
program was based upon materials in math and reading developed by 
the Southwest Regional Educational Laboratories and the SCOPE 
writing program (Gardner, 1979>. Ilateriais were criterion 
referenced with frequent mastery test. Tutoring took place for 
one to two hours per week, off-campus at libraries, and 
recreation centers . 

Tutors were recruited primarily from university departments 
of psvchoiogy and education. Tutors received several hours of 
training from the services coordinator in tutoring and in record 
keeping. Tutors were supervised for one hour each week in small 
groups. Tutors was closely monitored to insure that they met 
with the tutees as scheduled, followed up on tutee absences and 
kept careful records of tutee achievement. Most of the time of 
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the fuil"time services coordinator j^as^spent in monitoring tutor 
performance. Data on contact hours are given below. 

PARENT EDUCATION . lionthly v*orkshops for parents focused 
upon educat tona 1 I y relevant topics such as careers, how to help 
children study, and college entrance requirements. Not all 
parents attended even one of the workshops although some attended 
regularly. In addition, a newsletter was published quarterly 
and disseminated to parents, teachers, counselors, and other 
school personnel about the program. The newsletter included 
material similar to the parent workshops. 

Family involvement is a significant variable in academic 
attainm*?nt (Bear. Hess ar.d Shipman, 1966; Dowling. 1982? 
Filipczaky Lordeman and Friedman, 1977; Hoss, 1967; Scout and 
Borders. 1979). This constryct includes family attitudes toward 
education, parental or sibling skills at encouraging achievement, 
and family behaviors which support (or detract from) achievement. 
Family involvement was expected to be influenced by family 
acculturation. The program included children rrom a large 
proportion of first and second generation immigrant families 
reside in the Long Beach area, principally from Mexico and from 
Southeast Asia. Families presen-ted a great variety of attitudes 
and values toward education. 

OUTCOMES . The program services describeci abcve were 
intended to produce both proximal and distal changes in tutees 
and families which would be conducive to educational achievement. 
Identification wfth a higher achieving role model, the tutor, was 
hoped to impact achievement motivation. Success experiences for 
the tutee on some of the mastery tests was also hoped to 
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contribute to increased academic se I f -concept . It was hoped that 
family involvement could be increased diroctly through the family 
workshops as well as indirectly by the increased enthusiasm on 
the part of the children. 

The program services* interacting with the client/ramily 
impacts* were hoped to produce achievement gains in the tutees 
which would be measurable in several ways. Four achievement 
measures were used: (i) progress through the tutorial materials* 
C2) pre-to'post test gains on criterion referenced tests, (3) 
schoo 1 -admini stered standardized achievecnsnt tests, and (4) 
school grades. 
EVALUATION QUESTIONS 

CROSSROADS evaluation efforts had several purposes 
including evaiaa^mg actainmenc of process ana produc" 
objectives. Process objectives were generally net in a very 
timely and ccmpiete mariner. Educational materials which resulted 
from CROSSROADS included a tutoring manual for out -of -schoo I 
programs. However, attainment ot process and product objectives 
are necessary but insufficient conditions for the success of 
remedial educational interventions. The key evaluation question 
is whether educational achievement gains are measured. This 
report focuses on the the basic skill gains of the tuTiees. 
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METHODS - 

SUBJECTS 

Subjects were 166 tutees of CROSSROADS from four junior high 
schools in Long Beach* California. Schools chosen for CROSSROADS 
services were tour in the city whose students had the lowest 
standardized test scores. They served "transition" areas whose 
demographics were characterized by high family mobility Kdue to a 
Navy installation;, large ^jroportions of first and second 
generation families from Mexico and Southeast Asia, and a core- 
city area with characteristics such as deteriorating housing 
quality and low family incomes. To be eligible for CROSSROADS, 
children had to have a "C" or less average or score beiow the 
50th percentile in at least one area on the most recent 
sc ho ol" ad ministered stardardized achievement test. 

In response to a drop-out problem, the staff adopted early 
in the second program year, a criterion of eight (8) tutoring 
contacts as requisite before a tutee would be considered a 
"program tutee." Only "program tutees" are considered for the 
key evaluative analyses. A larger group of approximately 474 
children part^icipated to a lesser extent but moved from Long 
Beach, dropped out of CROSSROADS or were terminated before 
qualifying under the 8-session criterion. In the second program 
y??ar» the STiari made extra efforts to prevent dr opp ing-out . This 
was achi eved by requiring great er coram i tment from chil dren and 
parents before tutoring began. Parents were required to return 
consent forms and to telephone CROSSROADS. These efforts 
succeeded at reducing dropping-out to a low level. However, 
program tutees were clearly a select group of children. 
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MEASURES 

PROGRES S . The tutorial materials were very structured with 
frequent mastery tests embedded within them. Each level of the 
materials contained 12 to 20 units or steps, each wi\_h a mastery 
assessment (including parallel rorms for multiple tries). Each 
mastery test covered only one fundamental skill (e.g., blending 
consonants) and included from six to -^.hirty items which assessed 
the skill. There was a total of 12 levels in the three areas* 
reading, writing, and nath. 

The program used an 8076 pass criterion for these mastery 
tests; tutees who failed to meet this criterion returned for more 
drill on the specific material and later took a parallel form of 
the mastery test. This repeated up to three times until the 80% 
cricerion was met oy eacn curee* This criterion was cnosen 
based upon the following logic. Mastery tests assessed only one 
fundamental skill. Correctly responding to the substantial 
majority of the items should reflect understanding of the 
essential skill and rule out the possibility of chance attainment 
of the criterion. One hundred percent mastery might be cost- 
inefficient since children may make errors due to carelessness 
even when they understand the principle and it would not be 
desirable to repeat the unit with such children. 

Because of these ''hurdles** which each tutee had to pass in 
order to progress through the materials, progress rates were an 
indication of how much and how rapidly tutees were acquiring 
basic skills. Progress rate was regarded as the achievement 
measure most similar to the content of the tutorial materials* 
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and hence, the most cri terion-ref ere-nced^ 

PRE-POST TESTS . Upon entering the program, tutees were 

given a battery of placement tests keyed to the tutorial 
'materials. In each subject area, tutees were placed ror 

tutoring at the lowest level at which they failed to show 

mastery, again using an 80% criterion. Upon completing a level 

of tutorial materials, cutees were tested using a post-test which 

was a parallel form of the assessment test, also keyed to the 

tutorial materials. The pre-post gains were also regarded as a 

criterion referenced achievement measure. 

STANDARDIZED TESTS . Data from school administered 

standardized achievement tests (the Comprehensive Test of Basic 
✓Skills) were gathered for several years prior to entry into the 

program and for one or two years (as available) subsequent to the 

receipt of tutorial services. 

SCHOOL GRADES . School grades were also gathered from 

schools for several prior years and for one or two years 

following receipt of services. 
DESIGN 

CRITERION REFERENCED MEASURES . Rates of progress through 
the tutoring materials constituted a useful measure of tutee 
progress and program efficiency. Prv'^gress rates were examined 
by comparing the rirst to the second year or the program. 

Tutees wei e placed into tutorial levels based upon placement 
tests; parallel forms i:ere administered upon completion of a 
level. Therefore the design for analyses of criterion-referenced 
tests was pre test-posttes t . 

STANDARDIZED TESTS a:-:D GRADES . Because of the high drop-out 

10 
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rate and the select nature of the ''prcgratn tutees»" any 
comparison between CROSSROADS tutees and children not enrolled in 
the program would be uninterpretab 1 e. To cope with this serious 
design problem, the plan tor evaluating standardized test and 
grade results took advantage of the fact that tutees were divided 
into three partially overlapping groups based upon tutoring areas 
(reading* math, and writing). Children tutored in one area were 
compared to children tutored in other areas. For example, in 
analyses of math achievement, math tutees were compared to 
children not tutored in math but tutored in reading and/or 
writing. For analyses of reading and language scores, children 
tutored in either reading or writing were compared to children 
tutored in math only because of the potential overlap of reading 
and writing errects. Chiiaren tutorea in reaaing were examinea 
for possible gains in both reading and language test scores as 
were children tutored in writing. Since ail treatment and 
comparison subjects were selected into CROSSROADS using the same 
criteria, the effect was to block on the variables causing 
selection into the program, presum^.bly motivational variables. 
Validity threats are discussed below. 
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RESULTS ' 

Results associated with four different types of dependent 
variables are reported here. Each dependent variable is a 
measure of academic achievement. 

TUTEE PROGRESS. Tutees received an average of 22.3 sessions 
of tutoring representing a period of contact between the program 
and tutees ranging from approximately 11 to 23 weeks. Tutees 
averaged completing 1.45 levels of criterion-referenced tutoring 
mater ia I s. 

The average rate of progress over the two years of the 
program was one level of materials per 15.4 sessions of tutoring. 
There was a substantial increase in the progress rate from the 
first to the second year of the program. In the first year, the 
average tutee required 20 sessions to complete one level. For 27 
tutees who entered in the second program year, the rate was one 
level for each 12.8 sessions. Therefore, program efficiency 
apparently increased from the first to the second year of the 
program. 

CR 1 TER I ON-REFERENCED TEST GAINS. The mean pretest score for 
tutees was 62.3 percent (standard deviat ion^iO. 8 percent). This 
figure includes only levels in which program tutees eventually 
r ece i ved tutor ing . Drop-outs and tutees who tested -out ^ of a 
ievei oy scoring above '':he 80 percent criterion were excluded. 
The mean posttest score on the cr i ter ion tests was 34 percent 
(standard dev iat ion=10. 1 ) for program tutees. This represents a 
statistically significant gain (p < .01, one--tai i ed> - 

STANDARDIZED ACHIEVEMENT TESTS. Table 1 contains results 
from the standardized achievement -tests expressed as expanded 
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standard scores. This table shows contrasts between background 
test scores of children tutored and untutored in each tv^torial 



with the children tutored in math only in terms of reaiing and 
language test scores, I r, addition, two contrasts cornbin»=' 
children tutored in reading with those tutored in writing for 
analyses of reading and language scopes. 

Background contrasts suggest that the criterion referenced 
placement tests for reading and writing were sensitive to reading 
achievement histories of the children. The contrasts for 
language scores do not indicate a similar sensitivity to language 
achi evemen t histories. 

Table 2 shows the parallel results for followup scores. The 
pattern is similar witn tne tutored group having generally lower 
achievement than tha untutored group. For math, the gap has 
widened, to a statistically significant level. For children 
tutored in writing, the gap in reading scores has narrowed to a 
nons i gni f leant 1 eve 1 . 

Table 3 compares background and followup scores expressed as 
expanded standard scores. Changes for all groups, tutored and 
untutored are statistically significant. This pattern of 
findings creates doubt that the program is responsible for gains 
over time observed. To further examine this, additional analyses 
were conducted. First, homogeneity of the within group 
regressions was examined. Table 4 presents the results of 
regrirssing followup expanded standard test scores on background 
scores for tutored and untutored groups separately, Tutees were 



ar ea • 



Groups tutored in reading and writing were contrasted 
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grouped in severai ways for these ajiaXysjes based upon whether 
they were tutored in math, reading, or writing. Uns tandar di zed 
betas and y-intercepts differ for comparisons involving 
language scores. Betas differ for one comparison involving 
reading scores and the comparison of the y-lntercepts almost 
reaches significance at the five per cent level for this 
comparison as well. For these significant contrasts, the 
correlation coefficients and unstandardized betas are not 
consistently lower in the tutored groups as would be expected if 
tutoring reduced the strength of the association between 
background and fcllowup test scores. 

To further examine these findings, regression analyses were 
conducted to test for the magnitude and direction of the effects 
of tutoring (Table 5). Only one result approaches significance, 
that for math. This is in the direction opposite that predicted 
by the hypo thesis that the program was effective. 

Because the tests for homogeneity of within group 
regressions suggested differences in three sets of equations, 
additional tests were conducted. Untutored groups' regression 
aquations were used to predict posttest scores and residuals were 
examined (Tables 6 and 7), These indicate no significant 
differences in residual scores. 
SCHOOL GRADES 

Changes from background to follow-up were examined for 
school grades with no evidence of statistically significant 
changes in the grades of tutored or untutored children. Tutees 
averaged just below a ''C", 
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SUMMARY AND DISCl/SSJON 
The criterion referenced results from progress and pre- 
posttest measures shuw evidence that the tutees learned some 
•basic skillis. Criterion referenced materials were keyed to grade 
levels. Hence, for an average child to proceed through almost 
one and one half levels or mat.erials in twenty-two sessions is 
evidence that some skill acquisition is occurring. Simil^^'^ly, 
the pre-posttest gains indicate this. 

However, ther • ire two serious problems interpreting the 
cr i ter i on- r ef er enced results. One is that the simple pre-post 
design used with these data does not permit us to infsr whether 
the gains were due to program services; some or all of the 
learning might have occurred in school. Second and most 
seriously, the criterion referenced measured have an uncertain 
genera i i zab i I i ty to the domain of Dasic skills. The "criterion" 
to which the tutorial mastery tests were keyed was the content of 
the tutorial materials, not a more meaningful criterion. This is 
the weakness of criterion referenced measures^ the extent to 
which they reflect socially important amounts of learning is 
unknown. Given these drawbacks of the criterion referenced data, 
we muse turn to the test and grade data for evidence about the 
social significance of tutee gains and evidence that the program 
is responsible ror these gains. 

The school test results suggest that tutees improved from 
background to follow-up in all basic skills areas. The results 
of the regression and residual analyses provide, however, 
no support for the hypothesis that the program was responsible 
for the changes in these measures. Finally, grade data show no 
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evidence of improvement. 
VALIDIT Y ISSUES 

There are several questions about the validity or this 
evaluation of CROSSROADS. Since this design involves nonrandom 
assignment to treatment and comparison groups* regression effects 
must be considered. Procedures for selection into the program 
and placement in a tutorial area are key factors in evaluating 
the likelihood of regression effects. Children were eligible for 
the program if they had scored below the 50th percentile on the 
most recent standardized achievement test administration by the 
school district. However, once referred to the program, children 
were placed into a tutorial area based upon their performance on 
placement tests, not based upon their standardized test score 
below the SOtn percentile. Children were also eiigioie ir tney 
had grades betow a ''C, in a tutorial area; again, children were 
g.-^ouped on the basis of pretests, not grades. Therefore, for 
analyses involving test scores and grades, children were not 
grouped on the basis a measure used as a covariate. Regression 
effects due to testing error are therefore unlikely. 

Another potential validity threat is motivational 
differences between groups. Had the comparisons been between 
CROSSROADS tutees and untutored groups, this would have been a 
very serious problem. However, because both treatment and 
comparison children had to demonstrate similar motivation 
to enter the program? the design succeeds in blocking on 
this variable. 

More troublesome is the possibility of maturational effects 

ERLC i ^ 
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arising because skill development oocurs-at different rates in 
different children. If the tutees had been comparec. to a group 
of children with higher achieving backgrounds, this wouid have 
'been especially problematic. Since procedures for enrolling 
tutees created groups with similar overall achievement 
backgrounds, there is little reason to expect that one group 
would show a sudden acceleration of overall maturation. However, 
there is the possibility of ski 1 I --speci f ic maturation. Tutees 
were assigned to tutorial areas based upon placement cests which 
reflected a history of devel opmen t of a specific skill area. 
Sk i 1 I -spec i f i c maturation was examined with the tests for 
homogeneity of within-group regressions. For the one pair of 
groups which differed on the within-group regressions, skill 
specific maturation was examined with a test for the difference 
in residuals. These results suggest that sk i 1 I -speci f i c 
maturation is not concealing significant program effects. 

Another consideration is the possibility that effects of the 
progr am are general rather 'ihan s k i I I -spec i f i c by improving 
tutees study habits, academic self-concept or motivation, rather 
than by improving skills in a specific area. This would be 
consistent with the results showing improvements in test scores 
for both children tutored and those not tutored in each area. 
However, a competing explanation ror ^he overall i mprovem.int s in 
test scores is that the tutees who did not drop out represent a 
subset of tutees who are especially motivated to improve. 

Another methodological issue is connected with the use of 
standardised tests. A controversy exists about their usefulness 
as measures of the effectiveness of remedial educational 
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programs. There are two ^ajor probiems with the use of 
standardized tests for evaluation purposes. One is a validity 
issue and the other is a reliability issue. 

Floden. Porter. Schmidt, and Freeman (1980) analyzed several 
standardized educational tests and found surprisingly little 
overlap in content. This suggests that no one test adequately covers 
the domain of skill purportea to be assessed. Popham (i976> has 
pointed out that the procedures for standardization tend to 
remove items which are the core of educational curricula because 
these items are have low discr iminabi i i ty. Since they are the 
core of curricula, virtually all children learn them. Thus 
standardization results in test items peripheral to the curriculum. 

The reliability issue is also troublesome. For children 
scoring ac very iow percentiles, the numuex^ of items userui ror 
discriminating children is small because almost all children from 
these groups fail a large number of items. Guessing introduces a 
proportionately larger error component for lower scoring groups 
than for average scoring groups (Cole and Nitko, 1981). Fitz- 
Gibbons (1979) asserted that standardized tests are insensitive 
to the content of program curricula and usually understate gains 
achieved (cf. Popham, 1976). Weber (1978), however, in a 
critical review of "uses and abuses of standardized tests" 
indicated that such measures are usetul for program evaluation. 

Standardized achievement measures may be poor indicators of 
the impact of remedial programs. However, alternative 
indicators, cr i ter i on- r ef er enced measures, are also inadequate, 
because the lack of standardization makes their educational 
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significance uncertain. This probT^ffr'^ts** we 1 1 exemplified by the 
evaluation of CROSSROADS. The cr i ter i on- ref erenced results 
indicate that some learning took place but the standardized test 
results suggest that the magnitude of learning wr.s not of 
educational significance. We might conclude that the program was 
inefrective except than questions about the sensitivity of the 
measures have to be admitted to the discussion. The evaluator 
is left in the position of being unable to estimate with 
confidence educational impact at remedial levels. The strongest 
conclusion which is warranted is that the program was not 
effective enough to measurably impact the standardized tests. 
The residual uncertainty is unfair to programs which carry the 
burden of proof to show effectiveness and puts the evaluator in a 
questionable ethical position. 

Although a major undertaking, it is possible to construct a 
measure of basic educational skills with the advantager of 
criterion-referenced and standardized tests. Such a measure 
would be keyed to instructional materials used in classrooms; it 
would be standardized with one important change of procedure: 
item d i scr imi nab i I i ty would not be the principal basis for item 
T.nclusion. Items would be retained in proportions which reflect 
their prevalence and importance in curricula. The test would 
have one disadvantage: comparea to existing stanaaraizea 
measures, it ' M be lengthy. However, for evaluation purposes 
and perhaps for ndividual assessment, such a measure would be 
superior to any ex. Tting. Development of such a test should be a 
priority for the educational evaluation community. 

To summarize the methodological Issues, we may note that the 
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design succeeded in providing control for the mos't important 
competing hypotheses, regression, maturation, motivation, and 
selection. Results are thererore to be regarded as interpretab I e. 
However, because the design was quas i -exper i men ta i rather than 
experimental, competing hypotheses cannot be ruled out with 
absolute certainty. The principle caveat is that the potential 
i nsens i t i V i ty of the standardized tests creates the potential 
that the program had a modest impact which was not detectable 
with these measures. 
POLICY ISSUES 

CROSSROADS was originally intended to be a four year program 
but was terminated after two because of policy changes under 
President Reagan. A question arises about the appropriateness of a 
summative evaluation on tne program 5ince since it was rerminatea 
after less than its planned lifetime (see Benassi and Murray^ 
Note 1). 

It is certainly conceivable that, given two more years, 
the program could have improved. Therefore, this is a report on 
the program as implemented, not the program as conceived at the 
outset or as potentially developed. This is characteristic of 
evaluation; most programs are altered in the transition from the 
drawing board to implementation. The case of CROSSROADS is 
probably not an extreme example since slightly more than 
two years of service delivery took place, thanks primarily to 
diligence on the part of a competent staff who began providing 
services witnin a few weeks of funding. It is incumbent upon an 
evaluator not to ignore meaningful evidence even when programs 
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are terminated early. 11 — 

A distinct question is whether CROSSROADS existed long 
enough to deliver an adequate "dose"* of intervention. The 
average tutee was in contact with the program tor a period or 
less than 22 weeks, about one-fourth of the life of the program, 
in the second year. 27 tutees entered who eventually passed the 8 
session criterion for inclusion among "program tutees". These 
figures suggest that "program tutees" received an amount of 
tutoring which would have been typical of the larger group of 
tutees CROSSROADS might have served over four years. 

If we accept that it is appropriate to interpret the results 
from CROSSROADS, the important question remaining to be discussed 
then is why the program was of apparently limited effectiveness. 
The answer may lie in the out"of -schoc i model. The ou t-of -schoo I 
model is characterized by several problems which seeru inherent. 

The most significant problem is motivational. A much larger 
number of children were involved at a minimal level than were 
involved at a level which might reasonably have had educational 
benefits. This indicates that the out -of -schoo 1 approach has the 
formidable problem of recruitracmt and continuing the involvement 
of the tutees. This problem is much more significant for the 
out-of -schoo 1 model than for in-school approaches. Related 
factors which were more specific to CROSSROADS included tutee 
and . to some extent, tutor absences from schedu led tutoring 
sessions. In addition, the paid, individualized tutoring model 
is labor-intensive. These inefficiencies did not derive from 
lack of core staff competence. Staff were diligent, worked long 
hours, were well organized, and met process objectives in a 
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timely and tho ough manner. Rather* the inefficiencies seem to 
be inherent in the out-of -schoo I approach. 

A volunteer tutoring approach would reduce cost but not 
eliminate the problems ot drop-outs and absences. Creating a 
reward system for the tutees might reduce the problems or drop- 
outs and absences but might be costly. The Department of 
Education apparently believed that out-of school tutoring 
programs held promise. However, the experience of CROSSROADS 
seems to indicate that such approaches may find it very difficult 
to compete on cost and effici ency grounds with in-school mo dels. 

Although no single study can provide definitive conclusions 
about out-of -schoo I tutorial programs, this evaluation attempts 
to provide a contribution to the methodological issues in the 
area or evaluation of out-or-scnoo i tutoi-ing and to discussion or 
the policy issues associated with the national Basic Skills 
Improvement Program. 



2"^ 
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NOTE 

1. Benassi, v. and Murray, P. Comment on Dowe 1 1 ' s "Eva 1 uat i on 
or Gut-or -Schoo 1 Tutor ing" . Manuscript submitted to ERIC. 
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TABLP:^— _ . 

CONTRASTS BETWEEN TUTORED AND UNTUTORED GROUPS ON 
BACKGROUND TEST MEASURES (EXPANDED STANDARD SCORES) 



TUTORIAL 
AREA 



GROUP 



CRITERION NUMBER 
VARIABLE OF CASE5 



MEAN 



STANDARD 
DEVIATION 



READING 



UNTUTORED 
TUTORED 



READBKG* 



19 
82 



462. 9947 
*:;i6.359S 



78. 467 
62 . 464 



READING 



UNTUTORED 
TUTORED 



LANGBKG 



19 
82 



456. 2263 
^31 . 1378 



S3. 909 
81 . 466 



WRITING 



UNTUTORED 
TUTORED 



READBKG** 



19 

73 



462. 9947 
450. 6247 



78. 467 
54.632 



WRITING 



UNTUTORED 
TUTORED 



LANGBK 



19 

74 



456. 2263 
460. 2203 



63. 909 
66. 669 



READING OR 
WR I T I NG 



UNTUTORED 
TUTORED 



READBK* 



19 
118 



462. 9947 
430 . 7754 



78. 467 
62.085 



READING OR 
WRITING 



UNTUTORED 
TUTORED 



LANGBK 



19 
127 



519. 5737 
490. 8000 



67. 875 
63.468 



MATH 



UNTUTORED 

TUTOR EI' 



MATHBK 



28 

1 r-i^ 



450. 9929 
a a . ^ O o T 



65. 074 



*P<.05; **P<.01 

NOTE* The groups des - gnated as ''untutored" in reading and 
writing respectively included children who were tutored 
neither reading or language? that is, the untutored children 
these contrasts included children tutored only in math. 



reason for this is that the writing and reading placement 
were correlated. 



in 
in 
for 
The 
tests 



EVALUATION OF OUT-OF-SCHOOL TUTORING 



PAGE 25 



TABLE-^^^-iir'.., 



CONTRASTS BETWEEN TUTORED AND UNTUTORED GROUPS ON 
FOLLOWUP TEST MEASURES (EXPANDED STANDARD SCORES) 



TUTORIAL 


GROUP 


CRITERION.) 


NUMBER 






STANDARD 


AREA 




VARIABLE 


OF CASES 


MEAN 


DEVI ATI ON 


READING 


UNTUTORED 


READING* 


19 


519. 


5737 


67. 875 




TUTORED 




92 


^79. 


8457 


57. 660 


READING 


UNTUTORED 


LANGUAGE 


19 


526 . 


221 1 


65. 482 




TUTORED 




92 


504. 


3402 


61. 433 


WRITING 


UNTUTORED 


READING 


19 


519, 


5 737 


67. 875 




TUTORED 




82 


506. 


4524 


62.311 


WRITING 


UNTUTORED 


LANGUAGE 


19 


526. 


2211 


65. 482 




TUTORED 




82 


532. 


0293 


52.892 


READING OR 


UNTUTORED 


READING 


19 


456. 


2263 


63.909 


WRITING 


TUTORED 




118 


442, 


0407 


77. 601 


READING OR 


UNTUTORED 


LANGUAGE 


19 


526, 


2211 


65. 482 


WRITING 


TUTORED 




127 


515. 


2622 


61. 389 


MATH 


UNTUTORED 


MATH** 


32 


556. 


1406 


75. 452 




TUTORED 




114 


518. 


1263 


64.986 



♦P<.05? ♦♦P<.01 

N0TE2 The groups designated as ''untutored" in reading and in 
writing respectively included children who were tutored in 
neither reading nor language; that is, the untutored children for 
these contrasts included children tutored only in math. The 
reason for this is that the v-^riting and reading placement test's 
were correlated. 



EKLC 
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TABLE ^..^ . • 

BACKGROUND TO FOLLOWUP TEST SCbRE COMPARISONS 
FOR TUTORED AND UNTUTORED GROUPS 



TUTORIAL CRITERION NUMBER STANDARD 

.GROUP OF CASES MEAN DEVIATION 



READING 



TUTORED** 


READING 


BACKGROUND 


79 


418. 0215 


62. 065 






FOLLOWUP 


79 


477. 9215 


57. 511 


TUTORED** 


LANGUAGE 


BACKGROUND 


80 


432. 9075 


80. 594 






FOLLOWUP 


80 


502. 6250 


59. 055 


UNTUTORED** 


READING 


BACKGROUND 


52 


461 . 6519 


59.879 






FOLLOWUP 


52 


515. 2962 


64. 248 


UNTUTORED** 


LANGUAGE 


BACKGROUND 


52 


460. 8923 


62. 692 






FOLLOWUP 


52 


535. 2019 


55. 235 


WRITING 












TUTORED** 


READ I NG 


BACKGROUND 


69 


449. 0188 


54. 862 






FOLLOWUP 


69 


505. 5188 


58. 023 


TUTORED** 


LANGUAGE 


BACKGROUND 


70 


458. 6057 


67. 465 






FOLLOWUP 


70 


530. 7200 


49. 192 


UNTUTORED** 


READING 


BACKGROUND 


62 


420. 11 77 


71 . 411 






FOLLOWUP 


62 


478. 5548 


65. 240 


'JM'T'l.fTORED* ♦ 


LANGUAGE 


B AC KGROL'ND 


62 


Zi 2 ^ . 36^5 


80 . I 






FOLLOWUP 


62 


498. 2274 


65 . 646 


READING OR i 


WRITING 










TUTORED** 


READING 


BACKGROUND 


112 


430. 6491 


61 . 159 






FOLLOWUP 


112 


488. 2080 


61. 027 


TUTORED** 


LANGUAGE 


BACKGROUND 


113 


44 1 . 86^6 


76. 646 






FOLLOWUP 


113 


513. 6487 


58. 61? 


UNTUTORED** 


READING 


BACKGROUND 


19 


462. 9947 


78. 46*. 






FOLLOWUP 


19 


519. 5737 


67. 875 


UNTUTORED** 


LANGUAGE 


BACKGROUND 


19 


456. 2263 


63. 909 






FOLLOWUP 


19 


526.2211 


65. 482 


MATH 












TUTORED** 


HATH 


BACKGROUND 


103 


<*39. 8262 


75 . 073 






FOLLOWUP 


103 


516. 6155 




UNTUTORED** 


MATH 


BACKGROUND 


28 


^50. 9929 


65 . 07a 






FOLLOWUP 


28 


543.2393 


69. 442 
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T.ABL£_3J:^^- ^ 

SUMMARY OF THE REGRESSION DISCONTINUITY ANALYSES 



TUTORIAL 
AREA 



DEPENDENT 
VARIABLE 



READING 

TUTORED LANGUAGE 
UNTUTORED LANGUAGE 



CORR. 
COEFF. 



.6758 

.6361 



UNSTANDARDIZED 
BETA STD. ERROR 
OF BETA 



495* 
652* 



061 
192 



COEFFICIENTS 
CONSTANT STD. ERROR 
(Y-5NT) OF CONSTANT 

288.262* 26.924 
228.888* 88.294 



READING 
TUTORED 
UNTUTORED 



READING 
READING 



5672 
8451 



526* 
731* 



. 087 
. 112 



258. 235 
18i. 121 



36. 750 
52. 633 



MATH 

TUTORED MATH 
UNTUTORED MATH 



WRITING 
TUTORED 
UNTUTORED 

WRITING 
TUTORED 
UNTUTORED 



READING 
READING 



LANGUAGE 
LANGUAGE 



6094 
5228 



6917 
8451 



7267 
6361 



526 
558 



732 
731 



530* 
652* 



.068 
. 178 



. 093 
. 112 



061 
192 



285.171 30.394 
291.527 81.261 



177. 035 
181. 121 



42. 206 
52. 533 



287. 713* 28. 151 
228.888* 88.294 



READING OR 
WRITING 

TUTORED READING 
UNTUTORED READING 



6659 
8451 



664 
731 



. 071 
. 112 



202. 065 
181. 121 



30. 871 
52. 633 



READING 
OR WRITING 
TUTORED LANGUAGE 
UNTUTORED LANGUAGE 



6930 
6361 



529* 
652* 



.052 
. 192 



280.083* 23.406 
228.888* 88.294 



NOTE: The groups designated as "untutored" in reading and in 
writing respectively included children who were tutored in 
neither reading nor language; that is, the untutored children for 
these contrasts included children tutored only in math. The 
reason for this is that the writing and reading placement tests 
were correlated. 
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TABLE"^^r :* 

REGRESSION ANALYSES USING EXPANDED STANDARD SCORES 



DEPENDENT 
VARIABLE 



PREDICTOR 



UNSTD. 
BETA 



STD. 
BETA 



F-VALUE SIG. 



MATH 



READING 



READING 



LANGUAGE 



LANGUAGE 



MATH 

BACKGROUND 
MATH 

TUTORING 

READING 
BACKGROUND 
READING 
TUTORING 

READING 
BACKGROUND 
WRITING 
TUTORING 

LANGUAGE 
BACKGROUND 
READING 
TUTORING 

LANGUAGE 
BACKGROUND 
WRITING 
TUTORING 



. 531 
-20. 689 

.581 
15. 527 

. 731 
3.833 

. 515 
11 . 592 

.533 
-3. 183 



533 
128 

.637 
. 100 

. 735 
. 026 

.661 
. 076 

.696 
. 024 



68. 421 
3.299 



100.240 
. 129 



000 
072 



65.233 .0001 
1.614 .207 



0001 
721 



75. 674 . 0001 

.990 .322 

81.133 .0001 

.104 .748 
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TABLE-ve--. - 

TEST OF THE DIFFERENCE OF READING RESIDUALS USING UNTUTORED GROUP 
PREDICTION EQUATION AND SEPARATE VARIANCE ESTIMATES 



GROUP NUMBER MEAN STANDARD STANDARD 

OF CASES RESIDUAL DEVIATION ERROR 

UNTUTORED 

READING 52 -.1569 35.103 4.868 
TUTORED 

READING 79 1.6922 52.734 5.933 

T DFGREES OF 2-TAiL 

VALUE FREEDOM PROB. 

-.24 128.97 .810 
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TABLE 7 _ 

TEST OF THE DIFFERENCE OF LANGUAGE RESIDUALS USING, UNTUTORED GROUP 
PREDICTION EQUATION AND SEPARATE VARIANCE ESTIMATES 



GROUP 


NUMBER 
OF CASES 


MEAN 

RES I DUAL 


STANDARD 
DEV I AT I ON 


STANDARD 
ERROR 


UNTUTORED 
READING 80 


-8. 5187 


45. 328 


5. 068 


TUTORED 
READING 


19 


-. 1265 


50. 528 


11.592 


T 

VALUE 


DEGREES OF 
FREEDOM 


2-TAIL 
PROE. 






-.66 


25. 33 


. 513 






GROUP NUMBER 
OF CASES 

UNTUTORED 
READING OR 113 
WR I T I NG 


MEAN 

RESIDUAL 
-3.3350 


STANDARD 
DEVIATION 

43. 309 


STANDARD 
ERROR 

4. 074 


TUTORED 
READING 
WR I T I NG 


OR 19 


-. 1265 


50. 528 


11. 592 


T 

VALUE 


DEGREES OF 
FREEDOM 


2-TAIL 
PROS. 






-.26 


22.67 


. 796 







